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Benchmarking Terminology for Network Interconnection Devices 


Status of this Memo 


This memo provides information for the Internet community. It does 
not specify an Internet standard. Distribution of this memo is 
unlimited. 

Abstract 


This memo discusses and defines a number of terms that are used in 
describing performance benchmarking tests and the results of such 
tests. The terms defined in this memo will be used in additional 
memos to define specific benchmarking tests and the suggested format 
to be used in reporting the results of each of the tests. This memo 
is a product of the Benchmarking Methodology Working Group (BMWG) of 
the Internet Engineering Task Force (IETF). 


1. Introduction 


Vendors often engage in "specsmanship" in an attempt to give their 
products a better position in the marketplace. This usually involves 
much "smoke & mirrors" used to confuse the user. This memo and 
follow-up memos attempt to define a specific set of terminology and 
tests that vendors can use to measure and report the performance 


characteristics of network devices. This will provide the user 
comparable data from different vendors with which to evaluate these 
devices. 

2. Definition format 


Term to be defined. (e.g., Latency) 


Definition: 
The specific definition for the term. 


Discussion: 
A brief discussion about the term, it’s application 
and any restrictions on measurement procedures. 


Measurement units: 


The units used to report measurements of this 
term, if applicable. 
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Issues: 
List of issues or conditions that effect this term. 
See Also: 
List of other terms that are relevant to the discussion 
of this term. 
3. Term definitions 


3.1 Back-to-back 


Definition: 
Fixed length frames presented at a rate such that there 
is the minimum legal separation for a given medium 
between frames over a short to medium period of time, 
starting from an idle state. 


Discussion: 
A growing number of devices on a network can produce 
bursts of back-to-back frames. Remote disk servers 


using protocols like NFS, remote disk backup systems 
like rdump, and remote tape access systems can be 
configured such that a single request can result in 

a block of data being returned of as much as 64K octets. 
Over networks like ethernet with a relatively small MTU 
this results in many fragments to be transmitted. Since 
fragment reassembly will only be attempted if all 
fragments have been received, the loss of even one 
fragment because of the failure of some intermediate 
network device to process enough continuous frames can 
cause an endless loop as the sender repetitively 
attempts to send its large data block. 


With the increasing size of the Internet, routing 
updates can span many frames, with modern routers able 
to transmit very quickly. Missing frames of routing 
information can produce false indications of 
unreachability. Tests of this parameter are intended 
to determine the extent of data buffering in the 
device. 


Measurement units: 
Number of N-octet frames in burst. 


Issues: 


See Also: 
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3.2 Bridge 


Definition: 
A system which forwards data frames based on information 
in the data link layer. 


Discussion: 


Measurement units: 
n/a 


Issues: 


See Also: 
bridge/router (3.3) 
router (3.15) 


3.3 bridge/router 


Definition: 
A bridge/router is a network device that can selectively 


function as a router and/or a bridge based on the 
protocol of a specific frame. 


Discussion: 


Measurement units: 
n/a 


Issues: 


See Also: 
bridge (3.2) 
router (3.15) 


3.4 Constant Load 


Definition: 
Fixed length frames at a fixed interval time. 


Discussion: 
Although it is rare, to say the least, to encounter 
a steady state load on a network device in the real 
world, measurement of steady state performance may 
be useful in evaluating competing devices. The 
frame size is specified and constant. All device 
parameters are constant. When there is a checksum 
in the frame, it must be verified. 
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Measurement units: 
n/a 


Issues: 
unidirectional vs. bidirectional 


See Also: 
3.5 Data link frame size 


Definition: 
The number of octets in the frame from the first octet 
following the preamble to the end of the FCS, if 
present, or to the last octet of the data if there 
is no FCS. 


Discussion: 
There is much confusion in reporting the frame 
sizes used in testing network devices or network 
measurement. Some authors include the checksum, 
some do not. This is a specific definition for use 
in this and subsequent memos. 


Measurement units: 
octets 


Issues: 
See Also: 
3.6 Frame Loss Rate 


Definition: 
Percentage of frames that should have been forwarded 
by a network device under steady state (constant) 
load that were not forwarded due to lack of 
resources. 


Discussion: 
This measurement can be used in reporting the 
performance of a network device in an overloaded 
state. This can be a useful indication of how a 
device would perform under pathological network 
conditions such as broadcast storms. 


Measurement units: 


Percentage of N-octet offered frames that are dropped. 
To be reported as a graph of offered load vs frame loss. 
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Issues: 


See Also: 
overhead behavior (3.11) 
policy based filtering (3.13) 
MTU mismatch behavior (3.10) 


3.7 Inter Frame Gap 


Definition: 
The delay from the end of a data link frame as defined 
in section 3.5, to the start of the preamble of the 
next data link frame. 


Discussion: 
There is much confusion in reporting the between 
frame time used in testing network devices. This 
is a specific definition for use in this and subsequent 
memos. 


Measurement units: 
Time with fine enough units to distinguish between 
2 events. 


Issues: 
Link data rate. 


See Also: 
3.8 Latency 


Definition: 
For store and forward devices: 
The time interval starting when the last bit of the 
input frame reaches the input port and ending when 
the first bit of the output frame is seen on the 
output port. 


For bit forwarding devices: 

The time interval starting when the end of the first 
bit of the input frame reaches the input port and 
ending when the start of the first bit of the output 
frame is seen on the output port. 


Discussion: 
Variability of latency can be a problem. 
Some protocols are timing dependent (e.g., LAT and IPX). 
Future applications are likely to be sensitive to 
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network latency. Increased device delay can reduce 
the useful diameter of net. It is desired to 
eliminate the effect of the data rate on the latency 
measurement. This measurement should only reflect the 
actual within device latency. Measurements should be 
taken for a spectrum of frame sizes without changing 
the device setup. 


Ideally, the measurements for all devices would be from 
the first actual bit of the frame after the preamble. 
Theoretically a vendor could design a device that 
normally would be considered a store and forward 
device, a bridge for example, that begins transmitting 
a frame before it is fully received. This type of 
device is known as a "cut through" device. The 
assumption is that the device would somehow invalidate 
the partially transmitted frame if in receiving the 
remainder of the input frame, something came up that 
the frame or this specific forwarding of it was in 
error. For example, a bad checksum. In this case, 
the device would still be considered a store and 
forward device and the latency would still be 

from last bit in to first bit out, even though the 


value would be negative. The intent is to treat 
the device as a unit without regard to the internal 
structure. 


Measurement units: 
Time with fine enough units to distinguish between 
2 events. 


Issues: 


See Also: 
link speed mismatch (3.9) 
constant load (3.4) 
back-to-back (3.1) 
policy based filtering (3.13) 
single frame behavior (3.16) 


3.9 Link Speed Mismatch 


Definition: 
Speed mismatch between input and output data rates. 


Discussion: 
This does not refer to frame rate per se, it refers to 
the actual data rate of the data path. For example, 
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an Ethernet on one side and a 56KB serial link on the 
other. This is has also been referred to as the "fire 
hose effect". Networks that make use of serial links 
between local high speed networks will usually have 
link speed mismatch at each end of the serial links. 


Measurement units: 
Ratio of input and output data rates. 


Issues: 

See Also: 
constant load (3.4) 
back-to-back (3.1) 

3.10 MTU-mismatch behavior 

Definition: 
The network MTU (Maximum Transmission Unit) of the 
output network is smaller than the MTU of the input 
network, this results in fragmentation. 

Discussion: 
The performance of network devices can be significantly 


affected by having to fragment frames. 


Measurement units: 
Description of behavior. 


Issues: 
See Also: 
3.11 Overhead behavior 


Definition: 
Processing done other than that for normal data frames. 


Discussion: 
Network devices perform many functions in addition 
to forwarding frames. These tasks range from internal 


hardware testing to the processing of routing 
information and responding to network management 
requests. It is useful to know what the effect of 
these sorts of tasks is on the device performance. 
An example would be if a router were to suspend 
forwarding or accepting frames during the processing 
of large routing update for a complex protocol like 
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OSPF. It would be good to know of this sort of 
behavior. 


Measurement units: 
Any quantitative understanding of this behavior is by 
the determination of its effect on other measurements. 


Issues: 
bridging and routing protocols 
control processing 
icmp 
ip options processing 
fragmentation 
error processing 
event logging/statistics collection 
arp 


See Also: 
policy based filtering (3.13) 


3.12 Overloaded behavior 


Definition: 
When demand exceeds available system resources. 


Discussion: 
Devices in an overloaded state will lose frames. The 
device might lose frames that contain routing or 
configuration information. An overloaded state is 
assumed when there is any frame loss. 


Measurement units: 
Description of behavior of device in any overloaded 
states for both input and output overload conditions. 


Issues: 
How well does the device recover from overloaded state? 
How does source quench production effect device? 
What does device do when its resources are exhausted? 
What is response to system management in overloaded 
state? 

See Also: 


3.13 Policy based filtering 


Definition: 
Filtering is the process of discarding received 
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frames by administrative decision where normal 
operation would be to forward them. 


Discussion: 
Many network devices have the ability to be 
configured to discard frames based on a number 
of criteria. These criteria can range from simple 
source or destination addresses to examining 
specific fields in the data frame itself. 
Configuring many network devices to perform 
filtering operations impacts the throughput 
of the device. 


Measurement units: 


n/a 
Issues: 
flexibility of filter options 
number of filter conditions 
See Also: 


3.14 Restart behavior 


Definition: 
Reinitialization of system causing data loss. 


Discussion: 
During a period of time after a power up or 
reset, network devices do not accept and forward 
frames. The duration of this period of unavailability 
can be useful in evaluating devices. In addition, 
some network devices require some form of reset 
when specific setup variables are modified. If the 
reset period were long it might discourage network 
managers from modifying these variables on production 
networks. 


Measurement units: 
Description of device behavior under various restart 
conditions. 


Issues: 
Types: 
power on 
reload software image 
flush port, reset buffers 
restart current code image, without reconfuration 


Benchmarking Methodology Working Group [Page 9] 


RFC 1242 Benchmarking Terminology July 1991 


Under what conditions is a restart required? 
Does the device know when restart needed (i.e., hung 
state timeout)? 
Does the device recognize condition of too frequent 
auto-restart? 
Does the device run diagnostics on all or some resets? 
How may restart be initiated? 
physical intervention 
remote via terminal line or login over network 


See Also: 
3.15 Router 


Definition: 
A system which forwards data frames based on 
information in the network layer. 


Discussion: 
This implies "running" the network level protocol 
routing algorithm and performing whatever actions 
that the protocol requires. For example, decrementing 
the TTL field in the TCP/IP header. 


Measurement units: 
n/a 


Issues: 


See Also: 
bridge (3.2) 
bridge/router (3.3) 


3.16 Single frame behavior 


Definition: 
One frame received on the input to a device. 


Discussion: 
A data "stream" consisting of a single frame can 
require a network device to do a lot of processing. 
Figuring routes, performing ARPs, checking 
permissions etc., in general, setting up cache entries. 
Devices will often take much more time to process a 
single frame presented in isolation than it would if 
the same frame were part of a steady stream. There 
is a worry that some devices would even discard a single 
frame as part of the cache setup procedure under the 
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assumption that the frame is only the first of many. 


Measurement units: 
Description of the behavior of the device. 


Issues: 


See Also: 
policy based filtering (3.13) 


3.17 Throughput 


Definition: 
The maximum rate at which none of the offered frames 
are dropped by the device. 


Discussion: 
The throughput figure allows vendors to report a 
single value which has proven to have use in the 
marketplace. Since even the loss of one frame ina 
data stream can cause significant delays while 
waiting for the higher level protocols to time out, 
it is useful to know the actual maximum data 
rate that the device can support. Measurements should 
be taken over a assortment of frame sizes. Separate 
measurements for routed and bridged data in those 
devices that can support both. If there is a checksum 
in the received frame, full checksum processing must 
be done. 


Measurement units: 
N-octet input frames per second 
input bits per second 


Issues: 

single path vs. aggregate 

load 

unidirectional vs bidirectional 

checksum processing required on some protocols 
See Also: 


frame loss rate (3.6) 
constant load (3.4) 
back-to-back (3.1) 
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